<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
  <title>areaDetector Plugin NDPluginFile</title>
  <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type" />
</head>
<body>
  <div style="text-align: center">
    <h1>
      areaDetector Plugin NDPluginFile</h1>
    <h2>
      July 3, 2017</h2>
    <h2>
      Mark Rivers</h2>
    <h2>
      University of Chicago</h2>
  </div>
  <h2>
    Contents</h2>
  <ul>
    <li><a href="#Overview">Overview</a></li>
    <li><a href="NDFileJPEG.html">JPEG file plugin</a></li>
    <li><a href="NDFileTIFF.html">TIFF file plugin</a></li>
    <li><a href="NDFileMagick.html">GraphicsMagick file plugin</a></li>
    <li><a href="NDFileNetCDF.html">netCDF file plugin</a></li>
    <li><a href="NDFileNexus.html">NeXus (HDF) file plugin</a></li>
    <li><a href="NDFileHDF5.html">HDF5 file plugin</a></li>
    <li><a href="#Null">Null file plugin</a></li>
    <li><a href="#Performance">Performance</a></li>
  </ul>
  <h2 id="Overview">
    Overview
  </h2>
  <p>
    NDPluginFile is a base class from which actual file plugins are derived. There are
    currently file plugins for JPEG, TIFF, netCDF, Nexus, HDF5, and GraphicsMagick.
    The GraphicsMagick plugin can write a large number of formats, including JPEG, TIFF,
    PNG, PDF and many others.
  </p>
  <p>
    NDPluginFile inherits from NDPluginDriver. The <a href="areaDetectorDoxygenHTML/class_n_d_plugin_file.html">
      NDPluginFile class documentation</a> describes this class in detail. This class
    is designed to simplify the task of supporting a new file format. A derived class
    to support a new file format will typically need to implement only the pure virtual
    functions openFile(), readFile(), writeFile(), and closeFile(). Note that none of
    the current file plugins actually support the readFile() function yet, but this
    is planned for future releases.
  </p>
  <p>
    The NDArray callback data can be written to disk in 1 of 3 modes:
  </p>
  <ol>
    <li>Single mode. In this mode each NDArray callback results in a separate disk file.</li>
    <li>Capture mode. In this mode a memory buffer is allocated before saving begins.
      Callback arrays are placed into this buffer, and when capture stops the file is
      written to disk. This mode limits the number of frames that can be saved, because
      they all must fit in a memory buffer. It is the fastest mode, with the least probability
      of dropping arrays, because no disk I/O is required while capture is in progress.</li>
    <li>Stream mode. In this mode the data are written to a single disk file for those
      file formats that support multiple arrays per file (netCDF, NeXus and HDF5). Each
      frame is appended to the file without closing it. It is intermediate in speed between
      single mode and capture mode, but unlike capture mode it is not limited by available
      memory in the number of arrays that can be saved. For file formats that do not support
      multiple arrays per file (e.g. JPEG, TIFF) this mode is really the same as Single
      mode, except that one can specify a total number of files to save before stopping.</li>
  </ol>
  <p>
    The CreateDirectory record controls whether directories are created if they don't
    exist. If it is zero (default), no directories are created. If it is negative, then
    the absolute value is the maximum of directories that will be created (i.e. -1 will
    create a maximum of one directory to complete the path, -2 will create a maximum
    of 2 directories). If it is positive, then at least that many directories in the
    path must exist (i.e. a value of 1 will create all directories below the root directory
    and 2 will not create a directory in the root directory).
  </p>
  <p>
    If value of the LazyOpen record is "No" (0) then at least one array with the same
    datatype, array size, and attributes must have been collected by the driver (and/or
    plugins) from which the file saving plugin is getting its data <b>before</b> capture
    or stream mode file saving is started. This is required so that the openFile() function
    can know the dimensions and datatype of the arrays. If LazyOpen is "Yes" and the
    mode is Stream then the file open if deferred until the first array callback after
    streaming is started. This will slow down the saving of the first file.
  </p>
  <p>
    NDPluginFile supports all of the file saving parameters defined in <a href="areaDetectorDoc.html#asynNDArrayDriver">
      asynNDArrayDriver</a>, e.g. NDFilePath, NDFileName, etc. Thus, the same interface
    that is used for saving files directly in a driver is used for this plugin.
  </p>
  <p>
    The base class will delete the "original" file that the driver created for that
    array if the following are all true:</p>
  <ol>
    <li>The DeleteDriverFile record is "Yes".</li>
    <li>The file plugin has successfully written a new file.</li>
    <li>The array contains an attribute called "DriverFileName" that contains the full
      file name of the original file. The driver attributes XML file should contain a
      line like the following:<br />
      <tt>&lt;Attribute name="DriverFileName" type="PARAM" source="FULL_FILE_NAME" datatype="string"
        description="Driver file name"/&gt;</tt><br />
    </li>
  </ol>
  <p>
    The file saving plugins normally determine the name of the file from the FileName
    and FileNumber records. However, it is possible to have these values come instead
    from attributes in the array passed to the callback. The following 3 special attributes
    are used:</p>
  <ol>
    <li>FilePluginFileName: This attribute contains the file name.</li>
    <li>FilePluginFileNumber - This attribute contains the file number.</li>
    <li>FilePluginDestination - If this attribute contains the string "all" or the name
      of the asyn port for this plugin (e.g. FileTIFF1) then the plugin will write the
      array to a file. If this attribute has any other value then the plugin will ignore
      this array, and not write a file.</li>
  </ol>
  <p>
    Having the file information come from the array allows the driver to control which
    plugin saves a particular array. For example, there may be two file writing plugins
    active; the first saves the flat field files for a tomography experiment, and the
    second saves the normal projections. These plugins each stream data to a separate
    file. The driver knows which files are flat fields and which are normal projections,
    and adds the appropriate attributes to control which plugin saves each array. This
    would not be possible using a single plugin and EPICS PVs to switch the file, because
    of the problem of frames being buffered in the plugin queue.</p>
  <p>
    Normally files in Stream and Capture mode are automatically closed when the requested
    number of frames have been collected, or when the Capture record is set back to
    0. It is also possible to use an NDArray attribute to close the file. If the NDArray
    contains an attribute named FilePluginClose and the attribute value is non-zero
    then the current file will be closed.</p>
  <h2 id="Null">
    Null file plugin
  </h2>
  <p>
    NDFileNull inherits from NDPluginFile. This is a dummy file writer, which does not
    actually write anything. Its main purpose is to delete original driver files (DeleteDriverFile
    record) without writing the data to disk. For example, if the Pilatus is being used
    simply for alignment or testing one might want to display the images but then immediately
    delete the files that the Pilatus had created. This plugin allows one to do this,
    since it always reports success in "writing" the file, so the NDPluginFile base
    class will deleted the driver file if all the conditions outlined above are met.
  </p>
  <p>
    The <a href="areaDetectorDoxygenHTML/class_n_d_file_null.html">NDFileNull class documentation
    </a>describes this class in detail.
  </p>
  <p>
    The NDFileNull plugin is created with the NDFileNullConfigure command, either from
    C/C++ or from the EPICS IOC shell.</p>
  <pre>NDFileNullConfigure (const char *portName, int queueSize, int blockingCallbacks, 
                     const char *NDArrayPort, int NDArrayAddr, size_t maxMemory, 
                     int priority, int stackSize)
  </pre>
  <h2 id="Performance">
    Performance
  </h2>
  <p>
    Performance measurements were made on the Linux and Windows systems shown in the
    following table.</p>
  <table border="1" cellpadding="2" cellspacing="2" style="text-align: left">
    <tbody>
      <tr>
        <th>
          Model</th>
        <th>
          Operating system</th>
        <th>
          CPU</th>
        <th>
          RAM</th>
        <th>
          Disk</th>
      </tr>
      <tr>
        <td>
          Dell Precision T7500 </td>
        <td>
          Windows 7 64-bit </td>
        <td>
          Two quad-core Xeon X5647, 2.93 GHz (8 cores total) </td>
        <td>
          96 GB</td>
        <td>
          Two 500 GB 15K RPM SAS disks RAID 0 </td>
      </tr>
      <tr>
        <td>
          Penguin Relion 1751 Server</td>
        <td>
          Linux, Fedora Core 14 64-bit</td>
        <td>
          Two quad-core Xeon E5630, 2.53 GHz (8 cores total) </td>
        <td>
          12 GB</td>
        <td>
          Three 300 GB 15K RPM SAS disks No RAID</td>
      </tr>
    </tbody>
  </table>
  <p>
    The tests were performed under the following conditions:</p>
  <ul>
    <li>Simulation detector with a single small peak in Peaks mode. The driver was running
      in Continuous mode with AcquireTime=0, and AcquirePeriod=0, to achieve maximum frame
      rate. Both Windows and Linux were able to generate about 1500 frames/sec, which
      was faster than any of the file plugins could write, so they were always dropping
      frames.</li>
    <li>1024 x 1024 8-bit images, so each image was 1 MB.</li>
    <li>File plugins operating in non-blocking mode.</li>
    <li>Files were written to a local disk.</li>
    <li>For the file plugins that cannot save multiple images per file (JPEG, TIFF, Magick),
      tests were done only in Stream mode, which allows specifying the total number of
      files to write. For the drivers that can save multiple images per file (netCDF,
      Nexus, HDF5) tests were done first in Stream mode which saved 5000 images to a single
      5GB file. Tests were then done in Single mode. In this mode it is not possible to
      tell the file plugin to save exactly 5000 frames. Instead the simulation driver
      was put in Multiple mode with NumImages=5000. Thus the driver generated 5000 frames,
      but the plugin saved fewer than 5000 because it was dropping frames. The file rate
      was calculated as the number of frames actually written divided by the elapsed time.</li>
    <li>Note that the amount of data written to the files (5GB) was less than the amound
      of RAM on each system. It is thus possible that files are being cached in RAM, and
      that for more frames the I/O rate would drop. This is almost certainly true for
      the HDF5 file in stream mode on Windows, which has a data rate over 1GB/s.</li>
  </ul>
  <table border="1" cellpadding="2" cellspacing="2" style="text-align: left">
    <tbody>
      <tr>
        <th>
          File plugin</th>
        <th>
          Mode</th>
        <th>
          Windows (frames/s = MB/s)</th>
        <th>
          Linux (frames/s = MB/s)</th>
      </tr>
      <tr>
        <td>
          JPEG</td>
        <td>
          Stream</td>
        <td>
          79.1</td>
        <td>
          68.4</td>
      </tr>
      <tr>
        <td>
          TIFF</td>
        <td>
          Stream</td>
        <td>
          259.9</td>
        <td>
          186.4</td>
      </tr>
      <tr>
        <td>
          Magick, TIFF</td>
        <td>
          Stream</td>
        <td>
          55.7</td>
        <td>
          39.0</td>
      </tr>
      <tr>
        <td>
          netCDF</td>
        <td>
          Stream</td>
        <td>
          204.3</td>
        <td>
          185.0</td>
      </tr>
      <tr>
        <td>
          netCDF</td>
        <td>
          Single</td>
        <td>
          133.7</td>
        <td>
          179.1</td>
      </tr>
      <tr>
        <td>
          Nexus</td>
        <td>
          Stream</td>
        <td>
          192.3</td>
        <td>
          193.4</td>
      </tr>
      <tr>
        <td>
          Nexus</td>
        <td>
          Single</td>
        <td>
          106.3</td>
        <td>
          145.6</td>
      </tr>
      <tr>
        <td>
          HDF5</td>
        <td>
          Stream</td>
        <td>
          1113.5</td>
        <td>
          192.0</td>
      </tr>
      <tr>
        <td>
          HDF5</td>
        <td>
          Single</td>
        <td>
          139.7</td>
        <td>
          108.6</td>
      </tr>
    </tbody>
  </table>
</body>
</html>
